Heart rate (HR) monitoring is required in many real-world applications like healthcare, psychology understanding, affective computing and biometrics. Using only face videos for HR estimation provides an advantage of creating a non-invasive, inexpensive and unobtrusive system. However, HR estimation using face videos can be erroneous due to respiration, facial expressions, out-of-plane movements, camera parameters (like focus change) and environmental factors.